[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies by Gasoonjia · Pull Request #18760 · pytorch/executorch

Gasoonjia · 2026-04-08T05:31:46Z

Stack from ghstack (oldest at bottom):

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788
[ET Device Support] Extract shared device test utilities to reduce redundancy #18762
[ET Device Support] Add device tensor helper functions to TensorPtr API #18761
-> [ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
[ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
[ET Device Support] Define AOT device copy ops registry #18728
[ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
[ET Device Support] Module: allocate device memory for planned buffers #18476
[ET Device Support] MemoryManager: add per-buffer device metadata #18475
[ET Device Support] MethodMeta: expose per-buffer device placement API #18474
[ET Device Support] DeviceMemoryBuffer RAII class for device memory lifetime management #18473
[ET Device Support] Emitter reads non_const_buffer_device from graph meta #18472
[ET Device Support] Device-aware memory planning: separate buffers per device type #18375
[ET Device Support] Add NonConstBufferDevice schema for per-buffer device mapping #18330
[ET Device Support] DeviceAllocator interface and DeviceAllocatorRegistry #17535
[ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080
[ET Device Support] Parse device info from serialized tensor in tensor_parser #18328
[ET Device Support] Propagate device info from TensorSpec into serialized Tensor #18079
[executorch] Propagate device metadata from partitioner result onto TensorSpecs #18078

Add skip_h2d_for_method_inputs and skip_d2h_for_method_outputs config
flags to ExecutorchBackendConfig. These control whether PropagateDevicePass
skips inserting H2D/D2H copy ops at method I/O boundaries:

skip_h2d_for_method_inputs: user provides GPU tensor directly
skip_d2h_for_method_outputs: output stays on device for cross-method pipelines

Differential Revision: D99636778

…D/D2H copies Add skip_h2d_for_method_inputs and skip_d2h_for_method_outputs config flags to ExecutorchBackendConfig. These control whether PropagateDevicePass skips inserting H2D/D2H copy ops at method I/O boundaries: - skip_h2d_for_method_inputs: user provides GPU tensor directly - skip_d2h_for_method_outputs: output stays on device for cross-method pipelines Differential Revision: [D99636778](https://our.internmc.facebook.com/intern/diff/D99636778/) [ghstack-poisoned]

pytorch-bot · 2026-04-08T05:31:51Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18760

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 62 New Failures, 6 Unrelated Failures

As of commit 8e55dc6 with merge base b5ae0b9 ():

NEW FAILURES - The following jobs have failed:

Build Presets / apple (pybind) / build (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / windows (pybind) / build (gh)
Process completed with exit code 1.
periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 9f6afb996095afb3438ef2f26483283a0edac6da317bcac86e43179461d9f504 /exec failed with exit code 1
periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 8d1b7216b0580732b121d53b5c31ead267376b7b5fd41adea8cf180e4aa40b4b /exec failed with exit code 1
periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 9dcb441086973596cbaefbac275a64dab1d7a64327a7cd0b4ddab58ae2e74c60 /exec failed with exit code 1
pull / android / build-android (gh)
Process completed with exit code 2.
pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux-android / linux-job (gh)
RuntimeError: Command docker exec -t 1c100614d45e379c2105cd0a5868102636be5b257e8f8f77e83ff21b135224b4 /exec failed with exit code 2
pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-lora-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-lora-multimethod-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-mcu-cortex-m-backend / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t ba7c36ee987948555a49d1ce311c11cf23c553dbad463f52b9cc2b97a44260c8 /exec failed with exit code 1
pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t f7da07b01757e3fccbf7ca09511644f46fe524c9883599ba9c067ca55f3b6ec8 /exec failed with exit code 1
pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t cddc28bf64a67481105c4c4bab94ad0c7cb26e810ee52477716c1b61a4f0413c /exec failed with exit code 1
pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 7f5a4992ed9237cd9cb7b6088b6967803a8be2938c544509f64def36a98943b3 /exec failed with exit code 1
pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t f06aec28fbb4cc900c79e5795f462e00040af647184db4adaae9eca188e7eb10 /exec failed with exit code 1
pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 660bb0ce950226bfa23a84ac308f227001e3f7fc3eae6b1f8217fdddde3408a8 /exec failed with exit code 1
pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 50925a866b127f10df414d6299aac385d0014691d451504a7b6c362db372412b /exec failed with exit code 1
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t e9285b21937a3dac968392040e4c45984583eeb956d1b985ff154cb6fa58031d /exec failed with exit code 1
pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t b977d86f1c25b90e3458b7c9335126e22825f8bb60184b097a6c50ee351039d6 /exec failed with exit code 1
pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 157358029f6b73762cff545394fb0bdbb8c183f5f4704339b7badde5ad90bcf0 /exec failed with exit code 1
pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 2d3d0d63d33f01a927d7ce3064f4226b17fc42b2aee2b3493db57813c66613c8 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 30bd9e505865f2e9f33ae9653e8d55cdd6a5a99b85f07d5ec378a9d0494fc891 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t f85f730930b4ddf0526d79437d024176d50ef0ea0c01d947e5a992ea3871431a /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t 50600fb78247ecf708493678d61984d358ac0018ea00f343a4c094c90428efbe /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t c7bc63bdbb68a37dd9350a5c0f5c1e5fdfa469ef9e7a607fdfed3f8a9fe6a4b7 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t cc0c0053556e098205d76ad1e477d3b408d65ece7ec88b5447d094fb8b03e60e /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t eb383b7f20cb60067d0eb9e333491203aeca2cb65c421d1a0001c791e9e11e0e /exec failed with exit code 1
pull / test-moshi-linux / linux-job (gh)
test_mimi
pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
RuntimeError: Command docker exec -t c18c3cef8f609775f0199ebed7de3197eeaef877549fcdfc11bc0491526ba702 /exec failed with exit code 1
pull / test-openvino-linux / linux-job (gh)
RuntimeError: Command docker exec -t 69b3fb47c54421a15cb8810325f311b1f8828b378fcf8b23edc0a8512a6105ec /exec failed with exit code 1
pull / test-parakeet-xnnpack-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-phi-3-mini-runner-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-vulkan-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t ca95343b38cf6c1ac0303709ea6c7a11328a425a1f925845fed06e681b1105b4 /exec failed with exit code 1
pull / test-vulkan-operators-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / unittest / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t e72c87cb7f1493decd4224423fd55e9e52d61614321359a0a337a9e29274f890 /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
RuntimeError: Command docker exec -t 13d1f6c39b9cee03ccf0a5269eb9c15e941665cca89ca93dea1c398e64edf658 /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 7306a11c7ba888af49efab8ffcb494483c7909ac65ec25d9cb9ec14cc17f645b /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-nxp-neutron / linux-job (gh)
RuntimeError: Command docker exec -t e5a8c218bb09fcb867c553822d069dbc5700310fb69c18e57bdf6fa4b14b95cc /exec failed with exit code 2
pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
RuntimeError: Command docker exec -t f06ed275899ef67c6fa2ed449d0ff2a2c7449fe560d6e1589168e61f7d653422 /exec failed with exit code 2
pull / unittest-wasm-bindings / linux-job (gh)
RuntimeError: Command docker exec -t efe9e42550f3d02534f52e6599e6fdda48f59cb6a8a03d8734ae1117c46c7441 /exec failed with exit code 2

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / test-arm-cortex-m-size-test (bare_metal) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-build-wasm-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
Test ARM Backend / test-arm / test-backend-linux (arm_vgf_fp, operators) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-04-08T05:32:30Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

digantdesai

Review automatically exported from Phabricator review in Meta.

Gasoonjia requested review from JacobSzwejbka and larryliu0820 as code owners April 8, 2026 05:31

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 8, 2026

meta-codesync Bot added fb-exported meta-exported labels Apr 8, 2026

Gasoonjia mentioned this pull request Apr 9, 2026

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788

Open

digantdesai approved these changes Apr 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies#18760

[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies#18760
Gasoonjia wants to merge 1 commit intogh/gasoonjia/161/basefrom
gh/gasoonjia/161/head

Gasoonjia commented Apr 8, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 8, 2026

Uh oh!

digantdesai left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Gasoonjia commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18760

❌ 62 New Failures, 6 Unrelated Failures

Uh oh!

github-actions Bot commented Apr 8, 2026

This PR needs a release notes: label

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Gasoonjia commented Apr 8, 2026 •

edited

Loading

pytorch-bot Bot commented Apr 8, 2026 •

edited

Loading

This PR needs a `release notes:` label